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Abstract 

A modified Reformed Teaching Observation Protocol (RTOP) (Piburn & 
Sawada, 2000) instrument was used to separately examine eight technology and 
engineering (T&E) educators’ teaching of science, and T&E content and 
practices, as called for by the Standards for Technological Literacy: Content for 
the Study of Technology (International Technology Education Association 
[ITEA/ITEEA], 2000/2002/2007) and the Next Generation Science Standards: 
For States, By States (NGSS Lead States, 2013). The modified RTOP presented 
in this article can help provide feedback to teachers at all grade levels 
concerning their reformed teaching of science content and practices, and T&E 
content and practices. The instrument achieved acceptable interrater reliability 
(> 80%) and was tested in a larger study (Love & Wells, 2017). Results revealed 
a significant difference among participants’ teaching of science and T&E 
content and practices according to a myriad of variables, such as years of 
teaching experience, years of experience teaching the Foundations of 
Technology (FoT) curriculum, length of the FoT curriculum training attended, 
and select preparation experiences. Research implications suggest that the 
instrument affords equal applicability for examining science educators’ teaching 
of T&E content and practices and informing pre- and in-service teacher 
preparation efforts by determining key factors that significantly influence 
educators’ teaching of these concepts. 

Keywords: Technology and engineering education, science education, 
pedagogical content knowledge (PCK), observation instrument 

The main goal of this research was to examine the teaching of science and 
technology and engineering (T&E) content and practices embedded within 
Foundations of Technology (FoT), an international T&E education curriculum. 
To investigate this, an observation instrument was needed to separately rate 
instructors’ teaching of science content and practices and instructors’ teaching of 
T&E content and practices. As a result, a reliable and practical observation 
instrument was developed to quantify the level at which P-12 educators teach 
these concepts. This instrument, modified from the widely used Reformed 
Teaching Observation Protocol (RTOP) (Piburn & Sawada, 2000) and a newly 
created rubric (Appendix B), helps raters provide more consistent scores for 
observed teaching practices while providing timely and detailed feedback for 
instructors to enhance their teaching of science and T&E content and practices. 
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This research initiates a baseline for effectiveness trials that are necessary to 
establish the reliability of an instrument for examining the extent to which 
educators are adequately teaching science and T&E concepts as mandated by 
both the Standards for Technological Literacy: Content for the Study of 
Technology (STL) (International Technology Education Association 
[1TEA/1TEEA], 2000/2002/2007) and the Next Generation Science Standards: 
For States, By States (NGSS) (NGSS Lead States, 2013). 

Requisite to discussing the instrumentation is establishing a clear definition 
of the term T&E education to alleviate commonly held misconceptions 
(International Technology and Engineering Educators Association [ITEEA], 
2017). “Technology education . . . provides an opportunity for students to learn 
about the processes and knowledge related to technology that are needed to 
solve problems and extend human capabilities” (ITEA/ITEEA, 2000/2002/2007, 
p. 242). Dugger and Naik (2001) further clarified that 

Technology education is concerned with the broad spectrum of technology, 
which encompasses, but is not limited to, such areas as: design, making, 
problem solving, technological systems, resources and materials, criteria 
and constraints, processes, controls, optimization and trade-offs, invention, 
and many other human topics dealing with human innovation, (p. 31) 

T&E education requires the use of various technologies, materials, and tools to 
develop engineering solutions, which is distinctly different from educational 
technology (ITEEA, 2017). Understanding the difference between T&E 
education and educational technologies is critical for recognizing and accurately 
recording T&E content and practices ratings when using the instrument 
presented in this article. 

The modified instrument in this study was developed at an important time 
when science education was experiencing major changes. With the NGSS 
mandating that science educators integrate engineering content and practices 
within their curricula, it posed some legitimate concerns, specifically how 
science educators’ would be evaluated on their teaching of engineering content 
and practices—an area in which they had limited exposure and were not 
adequately prepared to teach in alignment with the NGSS (Nadelson & Farmer, 
2012). NGSS later developed the Educators Evaluating the Quality of 
Instructional Products (EQuIP) rubric for measuring and providing constructive 
criterion-based feedback to better align lessons and units with the NGSS (NGSS 
Lead States, 2016). However, as we present in subsequent sections, the EQuIP 
rubric cannot be used to adequately examine educators’ separate teaching of 
science and T&E content and practices. Therefore, a more accurate and reliable 
instrument was needed. 
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Review of Literature 

The philosophical basis for this research is grounded in Shulman’s (1987) 
concept of pedagogical content knowledge (PCK). Since its conception, PCK 
has been the topic of many notable publications within science education (e.g., 
Abell, 2008; Baxter & Lederman, 1999; Gess-Newsome, 1999; Hume & Berry, 
2011; Loughran, Berry, & Mulhall, 2012). It has also been a controversial 
research topic, with science education experts questioning its existence 
(Settlage, 2013) and how accurately it can be examined (Gess-Newsome & 
Lederman, 1999). Despite these criticisms, research has verified that teachers 
possess varying degrees of PCK based upon experience and training (Shulman, 
2004; Williams & Lockley, 2012). Given that science educators are now 
expected to teach engineering content and practices, logic dictates that they must 
have adequate PCK to properly teach engineering concepts within the context of 
science education. As the science profession moves forward in preparing 
classroom teachers to address engineering in science education, a reliable 
instrument will be needed for examining the extent of science educators’ PCK in 
T&E separate from their PCK in science. This research presents an instrument 
and method to address that need. 

Numerous research efforts within science, technology, engineering, and 
mathematics (STEM) education have examined PCK using a variety of methods, 
such as classroom observations, interviews, multiple choice questionnaires, 
assessing student work, and analyzing think aloud teaching tasks (Love, 2013). 
Despite these findings, there is still no unified agreement among researchers on 
the best method or methods to assess PCK because of its complexity (Park, 

Jang, Chen, & Jung, 2011). The authors therefore found it necessary to analyze a 
multitude of studies from various STEM disciplines in order to construct viable 
instrumentation for investigating PCK in this study. 

Loughran, Mulhall, and Berry (2004) developed the content representation 
(CoRe) and pedagogical and professional-experience repertoire (PaP-eRs) 
instruments. The CoRe instrument captures teachers’ holistic PCK relative to a 
specific topic, and the PaP-eRs instrument offers a view into a teaching-learning 
situation in which the content shapes the pedagogy. They have been widely used 
in STEM education studies (e.g., Bertram & Loughran, 2012; Hume & Berry, 
2011; Rollnick, Mundalamo, & Booth, 2013; Williams, Eames, Hume, & 
Lockley, 2012; Williams & Lockley, 2012) but are still viewed as imperfect 
tools due to their topic specificity and time intensive nature. A rubric with three 
levels of proficiency to rate science teachers’ PCK from coded interview 
responses was created by Lee, Brown, Luft, and Roehrig (2007). Their rubric 
did not accurately reflect all essential elements of PCK nor did it provide much 
detail about the specific teaching of content and practices. Park et al. (2011) 
developed a similar rubric that was used to rate teachers’ PCK based on 
interview responses, lesson plans, and classroom observations. Like Loughran, 
Berry, and Mulhall (2007), they acknowledged that PCK is topic specific, but it 
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is not efficient to make a rubric for each specific science topic taught. Park et al. 
admitted that when using a rubric to assess PCK, raters have to make inferences 
from what the teacher says, believes, and does. They concluded that scoring 
training is an important yet time intensive component needed when using rubrics 
to rate teachers’ PCK. 

The Cognitively Activating (COACTIV) Instruction project (Kunter et al., 
2007) developed instrumentation to help distinguish between content knowledge 
and PCK. The COACTIV instrumentation consisted of 23 paper-and-pencil 
questions representing scenarios that a mathematics instructor may encounter 
with their students. PCK ratings from this instrument were limited to 
participants’ paper-and-pencil responses. Furthermore, Gumbo and Williams 
(2014) investigated the PCK of T&E educators by recording observations every 
5 minutes, which created a series of snapshots of PCK elements observed over 
the duration of a lesson. They triangulated these observation data with both 
interviews and a content analysis of course textbooks. Although effective, they 
acknowledged that use of this instrument may not be practical for school 
systems because it demands a significant amount of time to collect and analyze 
data. 

The most notable instrument considered by the authors was the EQuIP 
rubric (NGSS Lead States, 2016) because it was aimed toward measuring 
science educators’ alignment and overall quality of lessons and units in relation 
to the NGSS. However, it was determined that the EQuIP rubric was not suitable 
for this study due to its inability to delineate between the teaching of science 
content and practices and the teaching of engineering content and practices. 
Rather, the EQuIP rubric combined science and engineering content and 
practices by using and/or when addressing the teaching of science and 
engineering concepts (NGSS Lead States, 2016). The EQuIP rubric fails to hold 
instructors accountable for teaching science and engineering concepts 
adequately. If using such an instrument to rate observed instruction of both 
science and engineering concepts, teachers would either rate extremely high or 
extremely low depending on how the term and/or is interpreted by the rater. 

This is a major reason that the authors chose not to utilize the EQuIP rubric for 
this study, because it was pertinent to investigate the differences between 
teaching of engineering and science content and practices. The instrument that 
the authors found most suitable for this study was the RTOP (Piburn & Sawada, 
2000 ). 

The Reformed Teaching Observation Protocol 

The RTOP (Piburn & Sawada, 2000) could be easily modified to help 
delineate between the teaching of science and T&E content and practices. It also 
examined instructors’ reformed teaching methods, which aligned well with the 
NGSS's expectations for teaching scientific inquiry and engineering design. The 
RTOP is grounded in constructivism and was designed as an observational 
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instrument to measure reformed teaching of mathematics and science. Various 
studies (e.g., Nicholas & Lomas, 2009; Ogletree, 2007; Park et al., 2011) have 
used it in conjunction with other instruments to measure teachers’ PCK. 
Specifically, Nicholas and Lomas (2009) found that the fourth section of the 
RTOP was able to assess teachers’ understanding of key content and provide 
valuable insight about their PCK. Piburn and Sawada (2000) and Taylor et al. 
(2013) found the RTOP to be a reliable and valid instrument aligned with 
national mathematics and science standards documents (e.g., American 
Association for the Advancement of Science [AAAS], 1989, 1993; National 
Council of Teachers of Mathematics [NCTM], 1989, 1991, 1995; National 
Research Council [NRC], 1996). Because of its alignment with the NGSS’s 
recommendation for research-based instructional reform (Taylor et ah, 2013), 
the RTOP was deemed adequate to rate observations in this study. 

As previously discussed, although a variety of instruments have been used 
to evaluate the PCK of science and T&E teachers, none were comprehensive in 
their evaluation of teacher practices. Given the growing focus in P-12 education 
on integrating the content and practices from multiple disciplines within a single 
subject, an instrument robust enough to identify specific instructional areas 
requiring further pedagogical preparation would be an important mechanism for 
informing pre- and in-service teacher preparation programs on those experiences 
identified as necessary for preparing educators to teach cross-disciplinary STEM 
concepts. Hence, modifications to the RTOP instrument were warranted. The 
following research questions helped guide the instrument modifications and data 
collection. 

1. How accurately can differences among educators’ teaching of science 
content, science practices, T&E content, and T&E practices each be 
quantified with a practical and reliable observation instrument? 

2. To what extent does instructors’ effectiveness in teaching science 
content, science practices, T&E content, and T&E practices differ 
according to the type of teacher preparation completed and the amount 
of teaching experience? 


Methodology 

The methodology employed in this study was designed to examine the 
preparation factors influencing the teaching of science concepts embedded 
within the FoT curriculum. 1 The authors decided to examine the teaching of 
T&E educators, as opposed to science educators who were only tasked with 
teaching new content and practices within the past year. T&E educators have 
been expected to integrate science and engineering concepts since the release of 
the STL (ITEA/1TEEA, 2000/2002/2007) 16 years ago, making them a more 


1 Details regarding the method employed are those described in previously 
reported research (see Love & Wells, 2017). 
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viable population for testing the content and practice items of the instrument. 

The expectation was that T&E educators’ increased experience with teaching 
these concepts would provide more accurate ratings that could otherwise be 
impacted by recently being tasked with teaching new content and practices. 

Participants 

This study utilized the same pool of participants as described in Love & 
Wells (2017). First, an online survey collecting demographic and preparation 
data was sent to all FoT teachers within 12 county school systems of a mid- 
Atlantic state. Of the 55 survey respondents, eight were purposefully selected 
for the classroom observation portion, which utilized the modified RTOP. The 
purposeful selection ensured that a sample of teachers with varying levels of 
science and T&E preparation experiences were observed. The demographics and 
preparation experiences of the observed participants are reported in Table 1. 


Table 1 

Demographic Data for Observed Participants 


Part. 

Gen. 

Eth. 

Age 

Years of 
teaching 

Years 

teaching 

FoT 

Years 

teaching 

science 

T&E 

education 

certified 

FoT 

training 

Inter- 

Disciplinary 
STEM course 

T 1 

M 

C 

62 

30 

2 

0 

Yes 

.5 day 

No 

T 2 

M 

C 

47 

10 

8 

1 

Yes 

1 week 

Yes 

T 3 

F 

c 

24 

2 

2 

0 

Yes 

1 week 

No 

T 4 

M 

c 

47 

13 

5 

2 

Yes 

.5 day 

Yes 

T 5 

M 

c 

56 

33 

4 

0 

Yes 

.5 day 

Yes 

T 6 

M 

c 

61 

28 

10 

0 

Yes 

1 week 

No 

T 7 

M 

c 

59 

21 

6 

0 

Yes 

1 week 

Yes 

T 8 

M 

AA 

25 

3 

3 

0 

No 

1 week 

No 


Notes: Part. = Participant; Gen. = Gender; Eth. = Ethnicity; T = Teacher; 
M = Male; F = Female; C = Caucasian; AA = African American. 


The observed participants consisted of predominantly White males who 
were certified to teach T&E education. The mean age of the participants was 48 
years, and the average number of years teaching was 18. They had very little if 
any experience teaching science courses and attended some form of training to 
learn how to teach the FoT curriculum. About half of the participants reported 
taking a higher education course that discussed methods for teaching 
interdisciplinary STEM concepts (e.g., science and engineering). The 
demographic and preparation data from this research and the full study (Love & 
Wells, 2017) were consistent with studies examining T&E educators at a 
national level (Love, 2015; Ernst & Williams, 2015). 
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Modifying the RTOP and Developing the Rubric 

Subscale 4 regarding content was modified to accurately examine the 
content and practices (PCK) of science and T&E concepts. These modifications 
allowed the data to be consistent with the language and teaching strategies 
described in the NGSS. The RTOP was modified with help from content 
specialists who had expertise in teacher evaluation, science education, and T&E 
education. The adaptations began by duplicating Subscale 4 and creating two 
similar yet separate subscales: one to score teaching of T&E content and 
practices and one for science content and practices. The words subject matter 
and concepts were replaced with the term content (Appendix A) to better align 
with the NGSS. This made it easier for raters to distinguish between teachers’ 
content knowledge and pedagogical practices during observations. Additionally, 
a rubric for each of the Subscale 4 criteria was created to provide clarity 
regarding what was being observed and to help elicit more consistent ratings 
(Appendix B). The content specialists suggested modifications to the rubric 
using language and criteria similar to that provided in the training guide section 
of the RTOP Reference Manual (Piburn & Sawada, 2000). This rubric helped 
observers determine ratings more easily and consistently while also staying true 
to the criteria described in the original RTOP. 

Training and Interrater Reliability 

Nicholas and Lomas (2009) established that the use of the RTOP by one 
trained rater consistently provided a valid assessment of teaching practice in a 
single classroom observation (about an hour). Due to similar time and funding 
limitations as experienced by Nicholas and Lomas, one trained rater conducted 
observations and rated participating teachers in this study. To ensure RTOP 
rating accuracy and reliability prior to actual use in research observations, 
content experts were used in establishing an acceptable interrater reliability. 
Establishing interrater reliability was conducted through two RTOP usage 
sessions with two content specialists who had expertise in Integrative STEM 
Education (Wells, 2016) teaching practices. All of the raters completed the 
online RTOP tutorials (Buffalo State University of New York, 2007). Following 
completion of the online modules, raters were asked to use the modified RTOP 
to rate two video-recorded FoT lessons from the same units as those observed 
later during the data collection. 

In line with standard protocols for establishing interrater reliability, three 
rounds were needed to reach consensus with an acceptable level of reliability. 
The first round had each rater use 10% of the RTOP items to independently 
analyze the first FoT lesson video, which was followed by a second round using 
an additional 10% of the RTOP items. At the end of each round, arbitration 
among raters was conducted to compare, discuss, and justify differences in co¬ 
ratings. The same procedures were followed for analyzing the second FoT 
lesson video, which was followed by a third round using the remaining 80% of 
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the RTOP items. At this point, an acceptable level of interrater agreement, 
greater than 80% (Howell, 2007), had been achieved (see Table 2), resulting in a 
viable coding scheme for using the RTOP for scoring a FoT lesson. 

Table 2 

RTOP Interrater Reliability Percentage Established Among Raters 


Observation 1 Observation 2 


Round 

Rater 

1 

Rater 

2 

Rater 

3 

Total 

Rater 

1 

Rater 

2 

Rater 

3 

Total 

1 

80% 

60% 

40% 

60% 

100% 

57% 

57% 

71% 

2 

67% 

50% 

50% 

56% 

100% 

86% 

86% 

90% 

3 

- 

- 

- 

- 

86% 

90% 

71% 

83% 


Note: Total was the percent agreement among all raters by round. 


During the classroom observation, the researcher remained as unobtrusive 
as possible by taking notes on what they observed regarding the teaching of both 
science and T&E content and practices. To provide the most accurate RTOP 
ratings, it was determined that lessons should be recorded for later review. To do 
this, the teacher was given a lapel microphone, and the audio recording was 
linked to the researcher’s notes using the AudioNote software. This software 
allowed the researcher to click on any portion of their notes and play the 
corresponding audio recorded during that part of the observation. Immediately 
after the completion of each school visit, the researcher used the modified RTOP 
instrument to rate the overall teaching strategies that they observed during the 
lesson. To ensure the ratings were as accurate as possible, within 48 hours, the 
researcher reviewed the lesson audio and their corresponding notes to confirm or 
adjust the ratings. After all observations were completed, they were again 
reanalyzed all at once via the audio recordings and notes for consistency across 
observation ratings. 

In modifying the RTOP for use in assessing FoT teaching practices, there 
are recognized limitations that are worthy of mention. Specifically, observer 
ratings could only be provided based on criterion specified by the instrument. 
Although Nicholas and Lomas (2009) found the RTOP acceptable for a single 
observer to rate one lesson, the ratings in this study only reflect a snapshot of the 
teacher’s full range of instructional practices. When using the RTOP, ratings are 
dependent upon the rater’s knowledge of teaching both science and T&E content 
and practices. Additionally, the sample of observed teachers consisted of 
primarily White males, which presents a limitation considering the diversity in 
the actual population of T&E educators. However, it is noteworthy that the 
convenience sample used in this study is actually reflective of national T&E 
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educator characteristics (Love, 2015; Ernst & Williams, 2015). Accounting for 
recognized limitations, the modified RTOP provided a viable instrument for 
quantifying teacher levels of science and T&E PCK, which is discussed in the 
following section. 


Observations and Results 

Relationship Among Observations and Preparation Factors 

The sum of all seven RTOP category ratings for each participant ranged 
from 18 to 118 out of a possible 140 with a mean of 59.4 (42%). Specifically in 
terms of science content and practices, the mean rating for teaching of science 
content was 9.6 (48%), and the mean rating for teaching of science practices was 
5.8 (29%). Four T&E teachers scored 6 (30%) or lower on teaching of science 
content, and three received a score of 1 (5%) on the teaching of science 
practices. Conversely, these teachers scored higher in observed teaching of T&E 
content and practices, as demonstrated by their mean ratings of 13.6 (68%) for 
T&E content and 7.6 (38%) for T&E practices. When examining these scores in 
more detail, there were only three participants who scored 9 (45%) or lower out 
of a possible 20 for teaching T&E content, but three teachers scored a two 
(10%) or lower for teaching T&E practices (see Table 3). 

Table 3 

Participants ’ Observation Ratings: Scores According to RTOP Categories 


Part. 



Category and score 



Total 

LD&I 

SC 

T&E C 

SP 

T&E P 

Cl 

S/TR 

T 1 

2 

6 

7 

1 

1 

3 

3 

23 

T 2 

12 

9 

17 

6 

12 

12 

10 

78 

T 3 

2 

3 

7 

1 

1 

3 

3 

20 

T 4 

12 

17 

20 

10 

16 

13 

14 

102 

T 5 

15 

19 

20 

15 

16 

15 

18 

118 

T 6 

0 

3 

9 

1 

2 

1 

2 

18 

T7 

6 

14 

19 

7 

6 

7 

10 

69 

T 8 

4 

6 

10 

5 

7 

9 

6 

47 

Mean 

6.6 

9.6 

13.6 

5.8 

7.6 

7.9 

8.3 

59.4 


Notes: Part. = Participant; T = Teacher; LD&I = lesson design and 
implementation; SC = science content; T&E C = technology and engineering 
content; SP = science practices; T&E P = technology and engineering practices; 
Cl = communicative interactions; S/TR = student-teacher relationships. Scores 
for each category range from 0-20, with higher scores indicating a greater 
rating. 
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When analyzing the RTOP scores according to the type of teacher 
preparation experiences participants had, all three groups received their highest 
mean ratings in T&E content and their lowest in science practices. Further 
analysis based on preparation experiences indicated that the participant holding 
an engineering degree had the highest mean RTOP rating (102), whereas 
teachers who completed teacher preparation programs in disciplines outside of 
science and T&E education received the lowest mean score (48; see Table 4). 

Table 4 


Summary’ of RTOP Ratings According to Preparation 


Teacher 

preparation 

Total «(|i) 

SC(ji) 

T&E C(|i) 

SP(li) 

T&E P(|i) 

T&E teacher 
prep 

4(58) 

10.5 

13 

6 

6 

Non-T&E 
teacher prep 

3(48) 

6 

9 

2 

3 

Engineering 

prep 

1(102) 

17 

20 

10 

16 


Notes: SC = science content; T&E C = technology and engineering content; 

SP = science practices; T&E P = technology and engineering practices. Scores 
for each category range from 0-20, with higher scores indicating a greater 
rating. 

Similar findings emerged when analyzing the data according to teaching 
experience categories. Novice teachers (1-5 years) had the lowest mean RTOP 
rating (33.5), veteran teachers (16 or more years) had the next highest rating 
(57), and intermediate teachers (6-15 years) had the highest rating (90). Again, 
all groups recorded their highest mean ratings in T&E content and their lowest 
in science practices (see Table 5). 

Table 5 


Summary of RTOP Ratings According to Experience 


Experience 

level 

Total 

n(|i) 

SC(ji) 

T&E C(|i) 

SP(R) 

T&E P(|i) 

Novice 

2(33.5) 

4.5 

8.5 

3 

4 

Intermediate 

2(90) 

13 

18.5 

8 

14 

Veteran 

4(57) 

10.5 

14 

6 

6 


Notes: SC = science content; T&E C = technology and engineering content; 
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SP = science practices; T&E P = technology and engineering practices. Scores 
for each category range from 0-20, with higher scores indicating a greater 
rating. 

The breadth of experience levels represented by participating teachers 
resulted in a range of high and low ratings. Consistent across type of preparation 
and teaching experience were the low scores in teaching science practices and 
high scores in teaching T&E content (see Tables 4 and 5). Among all eight 
participants, only two (Teachers 4 and 5) received a perfect score on any item, 
which occurred in teaching T&E content. Veteran Teachers 1 and 6 earned two 
of the lower total cumulative ratings for all seven categories (18 and 23 
respectively) among all observed participants, whereas intermediate teachers 
posted some of the higher summative ratings (102, 78). Furthermore, two of the 
lowest ratings were received by veteran instructors with over 28 years of 
teaching experience. It is tempting to make inferences with respect to the 
pedagogical expectations of veteran teachers, and caution is needed here not to 
draw inaccurate conclusions. The data in this study were not large enough to 
draw such conclusions; therefore, further analysis was done in order to identify 
the types and amounts of preparation experiences that had significant positive 
correlations with observed teaching of content and practices (Love & Wells, 
2017). 


Conclusions and Implications 

The overarching goal of this research was to develop an instrument that 
could separately examine the teaching of science and T&E content and 
practices. Recognizing the limitations imposed by the number of participants 
and their demographics, our analysis would suggest that the modified RTOP, 
together with the criterion-based rubric, has merit for discerning the degree to 
which the preparation of T&E teachers provides them with sufficient science 
content and practices to teach the science inherent within lessons such as those 
for FoT. The data suggest that the instrument could serve as a reliable and 
feasible observation tool to help school systems better focus their professional 
development efforts. One of the benefits of the instrument is that it directly 
measures those instructional strategies observed, as opposed to test question 
responses, and this research provides some evidence to suggest that the modified 
RTOP has potential for use with other science or T&E lessons. Further research 
would be needed to substantiate this potential. 

Although observation of T&E participants was limited to a single 1-hour 
lesson, their limited formal preparation in science content and practices (Love, 
2015; Love & Wells, 2017), coupled with low ratings on teaching science 
concepts in the lessons (Table 3), suggests the need to investigate whether or not 
similar results would be found for science teachers attempting to teach T&E 
content and practices inherent within science lessons. Considering that pre- 
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service science teacher preparation programs prior to NGSS did not address 
engineering practices, such research might prove useful in efforts to enhance 
science educators’ effectiveness in teaching engineering concepts with the rigor 
needed for students to make higher level cross-disciplinary connections. 
Moreover, the ratings found in relation to preparation type (Table 4) and years 
of teaching experience (Table 5) further emphasize the importance of examining 
the amount and types of preparation factors that influence instructors’ teaching 
of science and T&E concepts. However, in spite of these findings, further 
research examining teacher preparation experiences that influence science 
educators’ teaching of T&E concepts is needed to ensure a more holistic 
preparation of those instructors implementing the NGSS. The instrument 
presented in this article provides a starting point for such research and the 
potential for highlighting explicit areas in which school systems should focus 
their professional development efforts. 

Our findings have implications for examining the science and T&E PCK of 
teacher educators who are preparing science and T&E educators. With the 
recently released NGSS and mandated integration of engineering concepts, few 
would argue that the science educators preparing tomorrow’s science teachers 
may benefit from professional development on how best to integrate engineering 
concepts within the science curriculum. One readily available solution is 
collaboration between the science and T&E education programs in preparing 
both science and T&E teachers with the PCK that they need to teach cross- 
disciplinary STEM concepts (Wells, 2008). 
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Appendix A 

Propositional Knowledge: Science Content 

6a. 

The lesson involved detailed explanations and examples about 
fundamental science content identified by the curriculum. 

0 1234 

7a. 

The lesson promoted strongly coherent conceptual understanding of 
science content. 

0 1234 

8a. 

The teacher had a solid grasp of the science content presented in the 
lesson. 

0 1234 

9a. 

Elements of scientific abstraction (e.g., symbolic representations, theory 
building) were encouraged when it was important to do so. 

0 1234 

10a. 

Connections with science content to other content disciplines and/or real 
world phenomena were explored and valued. 

0 1234 


Propositional Knowledge: T&E Content 


6b. 

The lesson involved detailed explanations and examples about 
fundamental T&E content identified by the curriculum. 

0 1234 

7b. 

The lesson promoted strongly coherent conceptual understanding of T&E 
content. 

01234 

8b. 

The teacher had a solid grasp of T&E content presented in the lesson. 

0 1234 

9b. 

Elements of T&E abstraction (e.g., symbolic representations, theory 
building) were encouraged when it was important to do so. 

0 1234 

10b. 

Connections with T&E content to other content disciplines and/or real 
world phenomena were explored and valued. 

01234 
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Procedural Knowledge: Science Practices 

11a. 

Students used a variety of means (simulations, drawings, graphs, concrete 
materials, manipulatives, etc.) to represent science phenomena. 

0 1234 

12a. 

Students made predictions, estimations and/or hypotheses about key 
science concepts, and devised means for testing them. 

0 1234 

13a. 

Students were actively engaged in thought-provoking activity that often 
involved the critical assessment of science procedures. 

0 1234 

14a. 

Students were reflective about their science learning. 

0 1234 

15a. 

Intellectual rigor, constructive criticism, and the challenging of scientific 
content were valued. 

0 1234 


Procedural Knowledge: T&E Practices 


lib. 

Students used a variety of means (models, prototypes, drawings, graphs, 
concrete materials, manipulatives, etc.) to represent T&E phenomena. 

0 12 3 4 

12b. 

Students made predictions, estimations and/or hypotheses about key T&E 
concepts, and devised means for testing them. 

0 12 34 

13b. 

Students were actively engaged in thought-provoking activity that often 
involved the critical assessment of T&E procedures. 

0 12 34 

14b. 

Students were reflective about their T&E learning. 

0 12 34 

15b. 

Intellectual rigor, constructive criticism, and the challenging of T&E 
content were valued. 

0 12 34 
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Appendix B 


Question # 

0 

i 

2 

3 

4 

6a & 6b 

Detailed 
explanations and 
examples 

Targeted 
fundamental 
content was 

never 

mentioned. 

Targeted 
fundamental 
content was 

rarely 

mentioned and 

was not a focal 
point of the 

lesson. 

The lesson had 
very little 
focus on 
targeted 
fundamental 

content. 

The lesson was 
focused to 

some extent on 

targeted 

fundamental 

content. 

The lesson was 
focused 

entirely around 
targeted 
fundamental 
content. 

7 a & 7b 

Coherent 

conceptual 

understanding 

Targeted 
content was 
not interrelated 
with any other 
concepts. 

Targeted 
content was 
vaguely 
interrelated 
with other 
concepts to 
rarely increase 
its meaning. 

Targeted 
content was 
vaguely 
interrelated 
with other 
concepts to 
sometimes 
increase its 
meaning. 

Targeted 
content was 
vaguely 
interrelated 
with other 
concepts to 
moderately 
increase its 
meaning. 

Targeted 
content was 
strongly 
interrelated 
with other 
concepts to 
greatly 
increase its 
meaning. 

8a & 8b 

Solid grasp of 
content 

No evidence 
that the teacher 
has a solid 
grasp of the 
content in the 
information 
presented to the 
class. 

Teacher rarely 
illustrates a 
solid grasp of 
the content in 
the information 
presented to the 
class. 

Teacher 
sometimes 
illustrates a 
solid grasp of 
the content in 
the information 
presented to the 
class. 

Teacher 

frequently 

illustrates a 
solid grasp of 
the content in 
the information 
presented to the 
class. 

Teacher 
regularly 
illustrates a 
solid grasp of 
the content in 
the information 
presented to the 
class. 

9a & 9b 

Elements of 
abstraction 

Relationships 
were never 
represented in 
abstract and/or 
symbolic ways 
when it was 
important to do 
so. 

Relationships 
were rarely 
represented in 
abstract and/or 
symbolic ways 
when it was 
important to do 
so. 

Relationships 

were 

sometimes 

represented in 
abstract and/or 
symbolic ways 
when it was 
important to do 
so. 

Relationships 
were often 
represented in 
abstract and/or 
symbolic ways 
when it was 
important to do 
so. 

Relationships 
were regularly 
represented in 
abstract and/or 
symbolic ways 
when it was 
important to do 
so. 

10a & 10b 

Connections to 
other 

disciplines/real 

world 

Targeted 
content was 

never 

connected with 

content across 
disciplines or 
with a real 
world 
application 
example. 

Targeted 
content was 

rarely 

connected with 

content across 
disciplines or a 
real world 
application 
example. 

Targeted 
content was 

sometimes 

connected with 
content across 
disciplines or 
included an 
example of a 
real world 
application. 

Targeted 
content was 

often 

connected with 

content across 
disciplines or 
included 2 
examples of 
real world 
applications. 

Targeted 
content was 
regularly 
connected with 

content across 
disciplines and 
included more 
than 2 
examples of 
real world 
applications. 
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Question # 

0 

i 

2 

3 


11a & 11b 

Means to 
represent 
phenomena 

Students never 

used a variety 

of practices 

(models, 

drawings, 

graphs, 

concrete 

materials, 

manipulatives, 

etc.) to 

represent 

targeted 

science 

phenomena. 

Students 

incompletely 

used a variety 

of practices 

(models, 

drawings, 

graphs, 

concrete 

materials, 

manipulatives, 

etc.) to 

represent 

targeted 

science 

phenomena. 

Students 
sometimes 
(once or twice) 

used a 
complete 

practice 

(models, 

drawings, 

graphs, 

concrete 

materials, 

manipulatives, 

etc.) to 

represent 

targeted 

science 

phenomena. 

Students often 

(three) used a 

variety of 

complete 

practices 

(models, 

drawings, 

graphs, 

concrete 

materials, 

manipulatives, 

etc.) to 

represent 

targeted 

science 

phenomena. 

Students 
consistently 
(four or more) 

used a variety 

of complete 

practices 

(models, 

drawings, 

graphs, 

concrete 

materials, 

manipulatives, 

etc.) to 

represent 

targeted 

science 

phenomena. 

12a & 12b 

Made Predictions/ 
Estimations/ 
Hypotheses 

Students were 
not led to state 
predictions, 
estimations, 
and/or 
hypotheses 
associated with 
the targeted 
content, and 
did not have to 
devise ways to 
test it. 

Students were 
vaguely led to 
state 

predictions, 
estimations, 
and/or 
hypotheses 
associated with 
the targeted 
content, and 
did not have to 
devise ways to 
test them. 

Students were 
clearly led to 
state 

predictions, 
estimations, 
and/or 
hypotheses 
associated with 
the targeted 
content, and 
did not have to 
devise ways to 
test them. 

Students were 
explicitly led 
to state 
predictions, 
estimations, 
and/or 
hypotheses 
associated with 
the targeted 
content, and 
devised ways 
to test each. 

Students were 
explicitly led 
to state 
predictions, 
estimations, 
and/or 
hypotheses 
associated with 
the targeted 
content, and 
devised 

several ways to 
test each. 

13a & 13b 

Actively engaged 
in thought- 
provoking activity 
and critical 
assessment 

Students were 
never involved 
with the 
investigation, 
nor engaged in 
thought- 
provoking 
activity leading 
to critical 
assessment of 
procedures. 

Students were 
involved with 
the 

investigation, 
but rarely 
engaged in 
thought- 
provoking 
activity leading 
to critical 
assessment of 
procedures. 

Students were 
involved with 
the 

investigation, 
and sometimes 
engaged in 
thought- 
provoking 
activity leading 
to critical 
assessment of 
procedures. 

Students were 
involved with 
the 

investigation, 
and often 
engaged in 
thought- 
provoking 
activity leading 
to critical 
assessment of 
procedures. 

Students were 
involved with 
the 

investigation, 
and regularly 
engaged in 
thought- 
provoking 
activity leading 
to critical 
assessment of 
procedures. 
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Question # 

0 

i 

2 

3 


14a & 14b 

Reflective 

Students were 

never 

reflective about 
their learning 
on the targeted 
content or 
concepts in 
tasks. 

Students were 
vaguely 
reflective about 
their learning 
on the targeted 
content or 
concepts in 
tasks with a 
vague prompt 
and 

inappropriate 

time allowed. 

Students were 
minimally 
reflective about 
their learning 
on the targeted 
content or 
concepts in 
tasks with a 
minimal 
prompt and 
minimal time 
allowed. 

Students were 

clearly 

reflective about 
their learning 
on the targeted 
content or 
concepts in 
tasks with a 
clear prompt 
and adequate 
time allowed. 

Students were 
reflective on 
multiple 
occasions 
about their 
learning on the 
targeted 
content or 
concepts in 
tasks with 
clear prompts 
and ample 
times. 

15a & 15b 

Intellectual 
rigor/criticism/cha 
llenging 

Teacher never 
allows ideas to 
be presented, 
challenged, or 
negotiated by 
students on the 
targeted 
content. 

Teacher rarely 
allows ideas to 
be presented, 
challenged, or 
negotiated by 
the students on 
the targeted 
content, but 
without 
evidence. 

Teacher 
sometimes 
allows some 
ideas to be 
presented, 
challenged, or 
negotiated by 
the students on 
the targeted 
content with 
very little 
accurate 

evidence. 

Teacher often 
allows a 
variety of 
ideas to be 
presented, 
challenged, or 
negotiated by 
the students on 
the targeted 
content with 
some accurate 

evidence. 

Teacher 
always allows 
a variety of 
ideas to be 
presented, 
challenged, or 
negotiated by 
the students on 
the targeted 
content with 
adequate and 
accurate 
evidence. 
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